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Objective:  The  purpose  of  our  work  was  to  develop  an  integrated  system  with  image  guidance  and 
deformation  simulation  for  the  purpose  of  accurate  needle  insertion. 

Methods:  We  designed  an  ultrasound-guided  needle  insertion  manipulator  and  physical  model  to  sim¬ 
ulate  liver  deformation.  We  carried  out  an  in  vivo  experiment  using  a  porcine  liver  to  verify  the 
effectiveness  of  our  manipulator  and  model. 

Results:  The  results  of  the  in  vivo  experiment  showed  that  the  needle  insertion  manipulator  accurately 
positions  the  needle  tip  into  the  target.  The  experimental  results  also  showed  that  the  liver  model 
accurately  reproduces  the  nonlinear  increase  of  force  upon  the  needle  during  insertion. 

Discussion:  Based  on  these  results,  it  is  suggested  that  the  needle  insertion  manipulator  and  the  physical 
liver  model  developed  and  validated  in  this  work  are  effective  for  accurate  needle  insertion. 

©  2009  Elsevier  Ltd.  All  rights  reserved. 


1.  Introduction 

As  a  cancer  treatment  method,  percutaneous  therapy  is  now 
a  focus  of  attention.  For  example,  percutaneous  ethanol  injection 
therapy  (PEIT)  and  radio  frequency  ablation  (RFA)  are  performed 
for  liver  cancer.  In  this  form  of  treatment,  cancer  cells  exist¬ 
ing  inside  the  organ  are  necrotized  by  the  delivery  of  a  needle 
tip  that  injects  either  ethanol  (PEIT)  or  ablates  (RFA).  Percuta¬ 
neous  therapy  has  become  a  major  trend  in  liver  cancer  treatment 
because  it  is  very  minimally  invasive  but  achieves  sufficient 
results  [1,2]. 

Medical  procedures  such  as  PEIT  and  RFA  require  the  insertion 
of  a  needle  into  a  specific  part  of  the  diseased  area.  In  all  cases, 
the  needle  tip  should  be  as  close  as  possible  to  the  center  of  the 
target  cancer.  In  recent  years,  research  and  development  has  been 
carried  out  on  surgical  robots  and  navigation  systems  for  minimally 
invasive  and  precise  surgery  [3,4].  Research  into  robotic  systems 
to  assist  needle  insertion  has  also  been  conducted  to  improve  the 
accuracy  of  needle  placement  and  expansion  of  the  approach  path 
[5,6], 
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However,  there  is  a  remaining  large  problem  in  regard  to  accu¬ 
rate  needle  insertion:  organ  deformation.  The  needle  insertion  site, 
in  the  case  of  the  liver  for  example,  is  very  soft,  and  it  is  easy  for 
the  force  of  the  needle  to  deform  the  tissues  and,  as  a  result,  for  the 
position  of  the  target  cancer  to  be  displaced.  Therefore,  when  the 
needle  is  inserted,  there  is  a  corresponding  risk  of  organ  deforma¬ 
tion  and  the  resulting  target  cancer  displacement.  Therefore,  it  is 
necessary  to  take  organ  deformation  into  account  and  make  a  plan 
for  insertion  position  and  orientation  that  will  allow  the  manipula¬ 
tor  to  guide  the  needle  tip  to  the  displaced  cancer.  To  make  this  plan, 
a  simulation  method  in  a  virtual  surgery  environment  reproduced 
with  physical  models  of  organs  may  be  used  to  predict  and  visualize 
the  organ  deformation.  The  purpose  of  our  study  was  to  use  such 
a  method  and  develop  an  integrated  robotic  system  with  image 
guidance  and  deformation  simulation  for  the  purpose  of  accurate 
needle  insertion. 

I  A.  Needle  insertion  manipulator 

There  can  be  some  problems  when  surgeons  manually  insert 
needles,  including  different  treatment  results  and  the  possibility  of 
a  complicating  disease  resulting  from  needle  insertion  into  large 
blood  vessels,  etc.  The  purpose  for  developing  a  needle  insertion 
robot  is  to  realize  accurate  needle  insertion  independent  of  surgeon 
skills. 
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Various  types  of  robotic  systems  to  assist  with  needle  insertion 
have  been  researched  [7-19].  Masamune  and  Fichtinger  et  al.  have 
investigated  a  robotic  system  guided  by  images  from  a  CT  scanner 
inside  a  gantry  [7,8].  Loser  et  al.  have  developed  a  needle  insertion 
system  in  which  the  needle  position  is  controlled  using  the  image  of 
CT  fluoroscopy  [9].  Yanof  et  al.  have  researched  a  CT-guided  robot 
for  interventional  procedures  with  preoperative  and  intraoperative 
planning  [10]. 

Many  researchers  have  also  developed  MRI-guided  needle 
insertion  manipulators  [11-15].  Masamune  et  al.  have  developed 
a  small  robot  system  in  which  an  MRI  machine  controls  needle 
orientation  in  a  narrow  gantry  space  [11].  Hashizume  et  al.  have 
developed  an  MR  image-guided  surgical  robotic  system  for  min¬ 
imally  invasive  laparoscopic  surgery,  including  needle  insertion 
manipulator  [12].  Krieger  et  al.  have  developed  a  remotely  actu¬ 
ated  manipulator  for  access  to  prostate  tissue  under  MR  imaging 
guidance  [13].  Tajima  et  al.  have  developed  an  MRI-compatible 
manipulator  for  heart  surgery  [14].  And,  our  group  also  has  devel¬ 
oped  an  MRI-compatible  needle  insertion  manipulator  with  a 
bending  joint  inside  organ  [15]. 

Ultrasound-guided  needle  insertion  manipulators  also  have 
been  developed  [16-19].  Hong  et  al.  have  researched  an 
ultrasound-guided  needle  insertion  robot  for  percutaneous  chole- 
cystostomy  [16].  Fichtinger  et  al.  have  researched  a  robot-assisted 
prostate  brachytherapy  system  based  on  transrectal  ultrasound 
[17].  Terayama  et  al.  have  researched  a  needle  insertion  system 
that  uses  a  curved  multi-tube  device  [18].  Finally,  Corral  et  al.  have 
researched  an  automatic  needle-placement  robot  for  needle  inser¬ 
tion  into  the  spine  [19]. 

We  have  used  ultrasound  equipment  in  this  research  to  visu¬ 
alize  the  position  of  the  target  cancer,  and  ultrasound  images 
guided  operation  of  the  needle  insertion  manipulator.  The  ultra¬ 
sound  images  usually  are  used  in  the  needle  insertion  process 
because  ultrasound  equipment  provides  real-time  imaging  of  inter¬ 
nal  organs  and  is  easily  set  up.  In  addition,  compared  with  CT 
and  MRI  equipment,  ultrasound  equipment  is  in  more  widespread 
use. 

The  needle  insertion  manipulator  itself  must  be  compact,  so  that 
it  does  not  interfere  with  the  surgeon’s  access  to  the  patient  dur¬ 
ing  an  operation.  In  addition,  the  manipulator  must  position  the 
tip  of  the  needle  accurately.  However,  the  ultrasound-guided  nee¬ 
dle  insertion  manipulators  in  related  work  are  somewhat  large, 
while  these  manipulator  realized  accurate  positioning  of  needle 
tip.  Then,  our  work  focused  on  the  development  of  a  compact  nee¬ 
dle  insertion  system.  Our  design  for  an  integrated  needle  insertion 
manipulator  with  an  ultrasound  probe  realized  to  be  compact.  It 
should  be  noted  that  Hong  et  al.  already  have  developed  an  inte¬ 
grated  needle  insertion  manipulator  with  two  degrees  of  freedom 
[16],  while  our  manipulator  has  three  degree  of  freedom.  This  addi¬ 
tional  degree  of  freedom  increases  the  flexibility  of  the  approach 
path  and  the  likeliness  of  avoiding  large  blood  vessels.  In  addi¬ 
tion,  the  novel  power-transmission  system  we  proposed,  provide 
that  the  manipulator  system  was  more  compact  than  the  related 
work. 

1.2.  Organ  modeling 

Physical  organ  modeling  recently  has  aroused  considerable 
attention,  [20-33].  For  example,  the  Physiome  Project  is  known 
worldwide  [21].  In  addition,  Miller  et  al.  have  presented  three- 
dimensional,  nonlinear,  viscoelastic  constitutive  models  for  the 
liver  and  kidney  [22],  and  for  brain  tissue  [23].  Conventional 
research  into  the  modeling  of  living  bodies  mainly  concerns  defor¬ 
mation  analysis  using  a  finite  element  method  (FEM)  for  surgical 
simulation  and  training.  For  example,  Tiller  et  al.  have  developed  a 
deformation  analysis  of  the  uterus  using  FEM  [24].  Alterovitz  et 


al.  have  studied  the  simulation  of  needle  insertion  for  prostate 
brachytherapy  [25,26].  Meanwhile,  DiMaio  and  Salcudean  have 
used  a  linear  elastic  material  model  to  illustrate  a  system  for  mea¬ 
suring  the  extent  of  planar  tissue  phantom  deformation  during 
needle  insertion  [27,28].  Moreover,  Sakuma  et  al.  have  created  an 
equation  that  combines  both  logarithmic  and  polynomial  strain 
energy  forms  of  a  porcine  liver  from  combined  compression  and 
elongation  tests  [29,30].  Salcudean  et  al.  have  designed  a  planning 
system  to  determine  the  optimized  insertion  angle  and  position 
using  a  linear  and  nonlinear  organ  model  [31,32].  Finally,  Schwartz 
et  al.  have  presented  a  viscoelastic  and  nonlinear  model  for  the 
simulation  of  needle  insertion  [33]. 

In  general,  organ  tissue  is  inhomogeneous  and  exhibits  non¬ 
linear,  anisotropic  elastic  and  viscous  behavior  [20].  In  particular, 
viscoelastic  and  nonlinear  properties  are  important  for  precise 
needle  insertion  into  the  liver  because  of  following  reasons;  the 
nonlinear  properties  affect  the  response  of  liver  deformation 
when  there  is  a  large  deformation  [34],  while  the  viscoelas¬ 
tic  properties  affect  the  dependence  of  liver  deformation  on 
insertion  velocity  [35].  Thus,  an  accurate  liver  model,  including  vis¬ 
coelastic  and  nonlinear  properties,  is  required  for  precise  needle 
insertion. 

Our  work  focused  on  the  development  of  a  model,  includ¬ 
ing  both  viscoelastic  and  nonlinear  properties,  for  the  simulation 
of  needle  insertion.  Schwartz  et  al.  already  have  presented  a 
model  including  viscoelastic  and  nonlinear  properties  [33].  How¬ 
ever,  the  material  properties  included  in  that  model  are  only 
coarse  approximations  of  the  real  material  properties  of  bio¬ 
logical  soft  tissues  while  the  model  can  simulate  deformation 
at  speeds  which  are  compatible  with  real-time  applications.  It 
means  that  the  material  model  was  not  constructed  referring  to 
the  measured  material  data  from  real  tissues.  The  originality  of 
our  modeling  was  to  measure  the  detailed  material  properties  of 
real  liver  tissues  about  viscoelasticity  and  nonlinearity.  We  have 
also  developed  FEM-based  liver  model  using  the  measured  mate¬ 
rial  properties  and  simulated  the  liver  deformation  during  needle 
insertion  [36-40]. 

1.3.  Objectives 

The  objectives  of  our  work  were  to  develop  an  integrated  robotic 
system  with  an  image-guided  needle  insertion  manipulator  and 
also  a  deformation  simulation  system  using  a  physical  model  of  a 
liver  for  the  path  planning  of  needle  insertion.  This  paper  presents 
our  ultrasound-guided  needle  insertion  manipulator  and  physical 
model  to  simulate  liver  deformation.  We  also  discuss  the  in  vivo 
experiment  we  conducted  to  verify  the  effectiveness  of  our  needle 
insertion  manipulator  and  liver  model. 

We  developed  2D  liver  model  and  provided  the  2D  deformation 
simulation  in  this  paper  while  a  3D  model  would  provide  more 
accurate  results.  The  manipulator  movement  we  developed  is  also 
limited  in  2D  plane  of  ultrasound  image.  It  is  assumed  that  2D  sim¬ 
ulation  and  manipulator  movement  is  enough  effective  to  realize 
the  accurate  needle  insertion  because  the  surgeon  usually  insert 
the  needle  using  only  2D  ultrasound  images.  In  addition,  the  2D 
model  is  possible  to  be  used  intraoperatively  while  3D  deforma¬ 
tion  simulation  brings  a  potentially  vast  increase  in  the  calculation 
time  required. 

The  rest  of  this  paper  is  organized  as  follows:  Section  2  presents 
the  development  of  our  ultrasound-guided  needle  insertion  manip¬ 
ulator.  Section  3  discusses  the  development  of  our  liver  model, 
including  the  material  properties  of  the  model  and  the  formula¬ 
tion  and  solution  of  the  FEM-based  model.  Section  4  discusses  the 
methodology  of  our  in  vivo  validation  experiment  and  its  results. 
Finally,  Section  5  presents  a  summary,  overall  conclusions,  and  a 
look  at  future  work. 


Y.  Kobayashi  et  al.  /  Computerized  Medical  Imaging  and  Graphics  34  (2010)  9-18 


11 


Fig.  1.  The  process  of  treatment  using  the  integrated  needle  insertion  manipulator,  (a)  process  (I)— (II)  and  (b)  process  (III)— (VI). 


2.  Ultrasound-guided  needle  insertion  manipulator 

2.1.  Design  and  operational  concepts 

Our  design  for  an  integrated  needle  insertion  manipulator  with 
an  ultrasound  probe  provides  compactness  and  high-accuracy  posi¬ 
tioning,  because  it  decreases  registration  error.  The  process  of 
treatment  using  the  integrated  needle  insertion  manipulator  is  as 
follows  (Fig.  1): 


(I)  The  needle  insertion  manipulator  is  combined  to  the  position 
setting  equipment  such  as  endoscope  holding  device.  3D  posi¬ 
tion  of  the  manipulator  is  flexibly  changed  and  fixed  at  any 
position  by  the  device. 

(II)  The  surgeon  moves  the  ultrasound  probe  by  hand  and  scans 
the  target  cancer.  Surgeon  set  the  ultrasound  probe  with  nee¬ 
dle  insertion  manipulator  at  an  adequate  position  to  visualize 
the  target  cancer  in  ultrasound  image.  In  this  process,  the  nav¬ 
igation  system  to  present  the  position  of  ultrasound  probe  and 
tumor,  which  is  reported  in  such  as  [41  ],  is  useful  for  the  sur¬ 
geon  to  set  the  probe.  We  will  integrate  such  navigation  system 
near  future. 

(III)  Needle  insertion  position  and  orientation  are  decided  by  the 
surgeon  from  the  position  of  the  target  cancer  displayed  in  the 
ultrasound  images.  In  this  process,  the  surgeon  then  uses  the 
GUI  (Graphical  User  Interface)  to  order  the  needle  insertion 
path  from  the  computer. 

(IV)  The  position  of  the  ultrasound  probe  with  needle  insertion 
manipulator  is  obtained  using  an  optical  3D  position  mea¬ 
surement  system.  Then,  the  position  is  registered  to  the 
preoperative  CT  or  MRI  diagnostic  image  of  liver.  We  do  not 
integrate  this  registration  process  into  our  system.  The  regis¬ 
tration  technique  reported  in  such  as  [41  ]  will  be  used  near 
future. 


(V)  The  physical  liver  model  is  developed  based  on  the  obtained 
images.  Then,  deformation  of  the  liver  during  needle  insertion 
is  simulated.  The  surgeon  optimizes  insertion  path  using  the 
simulation  results  of  deformation  (see  Section  3) 

(VI)  The  position  of  the  needle  is  set  at  the  insertion  path  deter¬ 
mined  in  (V)  by  the  manipulator.  The  needle  is  inserted  into 
the  target  cancer. 


2.2.  Power  transmission 

The  manipulator  tends  to  be  large  when  the  actuator  is  included 
in  the  manipulator  itself.  Then,  to  keep  the  manipulator  compact, 
the  actuator  should  be  outside  the  manipulator,  positioned  in  such 
a  way  that  it  still  can  effectively  transmit  power  to  the  manipulator. 
Since  the  actuator  is  positioned  at  a  distance  from  the  manipulator, 
it  also  is  at  a  distance  from  the  patient,  which  increases  safety.  We 
used  a  power-transmission  system  called  a  “Flexible  Rack”  in  our 
development  of  the  needle  insertion  manipulator.  Fig.  2  presents 
the  flexible  rack  and  guide  tube.  Since  the  rack  is  made  from  flexible 
material,  it  bends  easily  and  provides  power  transmission  via  a  flex¬ 
ible  path,  whereas  a  rack  made  from  metal  provides  only  a  straight 
pass.  The  flexible  rack  is  moved  inside  a  guide  tube  to  avoid  rack 
deflection  and  interference  with  other  parts  of  the  system. 

At  times,  however,  the  rack  does  deflect  and  buckle,  which 
causes  lost  motion  (Li  -  L0  in  Fig.  3)  and,  in  turn,  a  loss  in  power  and 
accuracy.  So,  to  ensure  accurate  movement,  a  method  to  respond  to 
the  lost  motion  is  required.  Fig.  3  displays  the  power-transmission 
mechanism  that  we  developed.  The  flexible  rack  is  pushed  and 
pulled  when  the  input  pinion  gear  (pinion  1 )  is  rotated  by  the  input 
actuator.  The  rack  then  rotates  the  output  pinion  gear  (pinion  2). 
The  output  pinion  gear  is  coupled  to  the  shaft  of  a  ball  screw,  and  the 
flange  of  the  ball  screw  is  translated  by  the  rotation  of  the  ball  screw. 
The  translation  of  the  flange  is  used  as  the  output  for  manipulator 
movement. 


Fig.  2.  Flexible-rack  system,  (a)  Flexible  rack  and  (b)  flexible  rack  inside  guide  tube.  The  flexible  rack  bends  easily  and  provides  power  transmission  via  a  flexible  path  since 
the  rack  is  made  from  flexible  material.  The  flexible  rack  is  moved  inside  a  guide  tube  to  avoid  rack  deflection  and  interference  with  other  parts  of  the  system. 
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Fig.  3.  Mechanism  of  power  transmission.  The  flexible  rack  is  pushed  and  pulled 
when  the  input  pinion  gear  (pinion  1 )  is  rotated  by  the  input  actuator.  The  rack  then 
rotates  the  output  pinion  gear  (pinion  2).  The  output  pinion  gear  is  coupled  to  the 
shaft  of  a  ball  screw,  and  the  flange  of  the  ball  screw  is  translated  by  the  rotation  of 
the  ball  screw.  The  translation  of  the  flange  is  used  as  the  output  for  manipulator 
movement. 


The  following  actions  are  countermeasures  to  the  lost  motion: 
(1)  sensing  the  output  rotational  movement  (90,  (2)  reducing  this 
movement,  and  (3 )  increasing  the  low  back  drivability  of  this  move¬ 
ment.  In  regard  to  (1),  the  rotation  of  the  output  pinion  gear  is 
measured  by  the  encoder  attached  to  the  shaft  of  the  ball  screw. 


The  position  of  output  flange  Lf  is  controlled  using  this  information. 
In  regard  to  action  (2),  the  ball  screw  part  reduces  the  movement  L0 
at  the  output  part  to  the  translation  of  the  flange  Lf.  The  ball  screw 
part  has  the  low  back  drivability  (3).  It  realizes  to  cut  off  the  exter¬ 
nal  force  to  the  flexible  rack,  which  is  easily  moved  by  an  external 
force.  In  short,  this  power-transmission  system  achieves  accurate 
positioning  independent  of  lost  motion. 

2.3.  Manipulator  mechanism 

The  developed  manipulator  is  shown  in  Fig.  4.  Fig.  4(a)  is  the 
translation  side,  and  Fig.  4(b)  is  the  slide-crank  side.  Our  manipu¬ 
lator  has  three  degrees  of  freedom  (DOF).  Fig.  5(a)  is  a  mechanical 
outline  drawing  of  our  manipulator  mechanism,  while  Fig.  5(b)  dis¬ 
plays  the  composition  of  the  DOFs  of  our  manipulator.  One  DOF  is 
for  “(A)  horizontal  positioning”,  which  decides  the  position  of  the 
manipulator  horizontal  direction  in  relation  to  the  cross-section  of 
the  ultrasound  image.  One  DOF  is  for  “(B)  posture”,  which  decides 
the  needle  insertion  angle.  Another  DOF  is  for  “(C)  needle  inser¬ 
tion”,  which  decides  needle  insertion  after  the  needle  position  and 
angle  have  been  determined. 

All  DOFs  of  the  manipulator  are  moved  by  the  translation  move¬ 
ment  generated  by  the  flexible-rack  power-transmission  system. 
Fig.  6  displays  link  mechanism  to  realize  each  DOF.  The  DOF  of  (A) 
horizontal  positioning  is  moved  when  both  translations  (I)  and  (II) 


Driving  unit 


Probe  fixed  plate 


(a)  Translation  side 


(b)  Slidc-crank  side. 


Fig.  4.  Overview  of  manipulator:  (a)  is  the  translation  side,  and  (b)  is  the  slide-crank  side.  This  manipulator  has  three  degrees  of  freedom  (DOF).  A  six-axis  force  sensor  is 
installed  in  the  needle  insertion  unit.  The  accuracy  of  this  manipulator  is  about  0.1  mm.  (a)  Translation  side  and  (b)  slide-crank  side. 


(C)  Needle 
insertion 

(B) 

(a)  Mechanical  outline  drawing. 


Fig.  5.  Mechanism  of  manipulator.  All  DOFs  of  the  manipulator  are  moved  by  the  translation  movement  (I)— (III)  generated  by  the  flexible-rack  power-transmission  system, 
(a)  Mechanical  outline  drawing  and  (b)  composition  of  DOF. 
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(A)  Horizontal  positioning  (B)  Posture  (C)  Needle  insertion 

Fig.  6.  Link  mechanism  to  realize  each  DOF.  The  DOF  of  (A)  horizontal  positioning  is  moved  when  both  translations  (I)  and  (II)  are  driven  by  the  same  displacement.  The 
DOF  of  (B)  posture  is  changed  only  when  translation  (II)  is  displaced  while  translation  (I)  is  fixed.  The  DOF  of  (C)  needle  insertion  is  driven  by  translation  (III).  (A)  Horizontal 
positioning,  (B)  posture  and  (C)  needle  insertion. 


are  driven  by  the  same  displacement.  The  DOF  of  (B)  Posture  is 
changed  only  when  translation  (II)  is  displaced  while  translation  (I) 
is  fixed.  In  this  situation,  the  translation  movement  generated  by 
the  flexible-rack  power-transmission  system  is  changed  to  rotation 
movement  using  the  slider-crank  mechanism  to  set  the  posture  of 
the  needle.  The  DOF  of  (C)  Needle  insertion  is  driven  by  translation 
(III);  the  needle  is  attached  to  the  unit  of  translation  (III),  and  it  is 
moved  in  the  axial  direction. 

The  range  of  movement  of  each  DOF  is  decided  in  response  to 
various  positions  of  the  target  cancer  in  both  shallow  and  deep 
locations  in  the  liver.  (A)  Horizontal  positioning  has  a  40-mm  trans¬ 
lation  of  movement,  (B)  Posture  has  a  ±30°  range  of  movement,  and 
(C)  Needle  insertion  has  a  60-mm  translation  of  movement.  A  six- 
axis  force  sensor  is  installed  in  the  needle  insertion  unit,  and  the 
force  information  loaded  on  the  needle  is  measured  by  the  sensor. 
The  manipulator,  which  is  designed  to  hold  various  sizes  of  con¬ 
ventional  ultrasound  probes,  is  80  mm  high  x  90  mm  wide  x  44  mm 
deep,  which  is  a  handy  size  for  use  in  operations.  The  position¬ 
ing  accuracy  of  this  manipulator  is  about  0.1  mm  from  the  result 
of  experiment  using  3D  position  measurement  system,  which  has 
about  0.1  mm  accuracy. 

3.  Deformation  simulation  of  liver 

Our  work  focused  on  the  development  of  a  model,  including  both 
viscoelastic  and  nonlinear  properties,  for  the  simulation  of  needle 
insertion.  A  porcine  liver  was  used  as  the  sample  for  this  study 
because  porcine  organs  have  frequently  been  used  as  a  substitute 
for  human  organs  due  to  their  similarity  [29,34]. 

3A.  Material  properties 

We  already  have  reported  the  material  properties  of  a  porcine 
liver  in  [36,37],  and  these  papers  also  gave  specific  descriptions  of 
the  physical  properties  of  the  liver.  Thus,  only  a  simplified  expla¬ 
nation  of  the  material  behavior  used  for  deformation  calculation  is 
shown  in  this  paper.  Experiments  were  individually  implemented 
to  measure  the  physical  properties  of  the  porcine’s  interior  liver 
using  a  rheometer  (TA-Instrument:  AR550).  The  shear  modulus, 
shear  stress,  and  shear  strain  were  then  calculated  based  on  these 
results. 

3AA.  Viscoelastic  properties 

A  dynamic  viscoelastic  test  was  carried  out  to  measure  the 
frequency  response  of  the  liver.  A  sine-wave  stress  from  0.1  to 
10  rad/s,  providing  3%  strain  amplitude,  was  loaded  on  the  liver, 
and  a  dynamic  viscoelastic  test  was  conducted.  The  mechanical 
impedance  of  the  porcine  liver  obtained  from  the  result  of  this 
test  is  shown  in  Fig.  7.  The  needle  is  generally  inserted  into  the 
organ  at  a  low  velocity;  hence  the  response  is  mainly  affected  by 


the  low  frequency  characteristics.  Thus,  we  used  the  viscoelastic 
model  using  the  fractional  derivative  described  in  (1),  which  takes 
only  low-frequency  characteristics  into  consideration. 


where  G  is  the  viscoelasticity,  k  is  the  order  of  derivative,  y  is  the 
shear  strain,  r  is  the  shear  stress  and  t  is  time.  The  derivative  order 
k  was  approximately  equal  to  0.1,  based  on  the  slopes  of  G'  and  G" 
shown  in  Fig.  7. 

3. 1 .2.  Nonlinear  properties  (strain  dependence  of  elastic 
modulus) 

The  nonlinear  characteristics  of  liver  as  a  material  were  inves¬ 
tigated  based  on  the  creep  test,  in  which  the  step  response  is 
measured.  The  steady  state  of  the  step  response  following  sufficient 
elapsed  time  exhibits  the  low-frequency  characteristics  described 
in  (1).  Eq.  (1)  becomes  (2)  if  (1)  is  solved  by  the  condition  of  the 
creep  test. 

r=mrnrf  =  **  <2> 

where  rc  is  constant  shear  stress,  r()  is  the  gamma  function,  and 
yc  is  the  coefficient  deciding  the  strain  value. 

The  creep  test  for  each  stress  was  carried  out  repeatedly  while 
the  viscoelasticity  G  and  strain  yc  for  each  stress  were  calculated 
using  (2).  From  these  results,  the  graph  in  Fig.  8  shows  the  depen¬ 
dence  of  viscoelasticity  G  on  strain  yc.  A  liver  with  a  low  strain  of  less 
than  about  0.35  displays  linear  characteristics  and  a  viscoelasticity 
G  at  a  constant  400  Pa.  A  liver  with  a  high  strain  of  more  than  about 


Fig.  7.  Mechanical  impedance  of  liver.  This  figure  shows  the  experimental  result 
of  dynamic  viscoelastic  test.  G*  is  a  complex  shear  modulus,  G'  is  a  storage  elastic 
modulus,  and  G”  is  a  loss  elastic  modulus.  G  and  G'  increase  proportionally  in  the 
log-log  diagram.  Then,  we  used  the  viscoelastic  model  using  the  fractional  derivative 
described  in  (1).  This  model  accurately  fit  the  experimental  result  as  shown  in  this 
figure. 
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Fig.  8.  Strain  dependence  of  viscoelasticity  G.  This  result  was  obtained  from  the 
creep  test  for  each  stress.  A  liver  with  a  low  strain  displays  linear  characteristics, 
while  a  liver  with  a  high  strain  displays  nonlinear  characteristics.  Then,  the  strain 
dependence  of  viscoelasticity  was  modeled  using  the  quadratic  function  of  strain 
shown  in  (3). 


0.35  displays  nonlinear  characteristics  and  an  increased  degree  of 
viscoelasticity  G.  Then,  the  strain  dependence  of  viscoelasticity  was 
modeled  using  the  quadratic  function  of  strain  shown  in  (3): 


G(y)  = 


Go  ( Y<Yo ) 

C0(l  +uY(y-  Yo)2)  (y  >  Yo) 


(3) 


where  G0  is  the  viscoelastic  modulus  of  the  strain  slope  between 
0  and  0.35  in  Fig.  8.  is  the  coefficient  deciding  the  change  of 
stiffness,  and  y0  is  the  strain  at  which  the  characteristics  of  the 
liver  change  to  show  nonlinearity. 


3.2.  FEM-based  modeling 

This  section  shows  the  formulation  of  the  FEM  model  using  the 
material  properties  described  in  Section  3.1.  Since  we  already  have 
given  specific  descriptions  in  this  regard  [38,39],  the  following  will 
be  only  a  simplified  explanation. 

The  expressions  between  the  displacements  at  all  the  nodal 
points  and  all  the  applied  loads  are  shown  in  (9)— ( 11),  from  the 
result  of  (8). 

I<(U)DWU  =  F  (9) 

where  F  is  external  force  vector,  K(U)  is  nonlinear  overall  stiffness 
matrix,  U  is  overall  displacement  (deformation)  vector,  and 
means  kth-order  derivative. 

K(U)=  Y,  k <*)  (10) 

all  elements 

!«:>«:!  <n> 

where  k  is  the  nonlinear  element  stiffness  matrix,  ko  is  the  element 
stiffness  matrix  when  the  liver  tissue  shows  linear  characteristics, 
as  is  the  coefficient  deciding  the  change  of  stiffness,  and  £r  is  the 
relative  strain. 

First,  the  solution  for  the  viscoelastic  system  is  shown  in  Section 

3.2.1,  and  the  solution  for  the  nonlinear  system  is  given  in  Section 

3.2.2.  Finally,  the  solution  for  (9)  is  shown  based  on  the  discussions 
from  both  Sections  3.2.1  and  3.2.2. 


3.1.3.  Shear  stress-strain  relation 

The  material  properties  of  the  liver  were  modeled  using  (4)  from 
the  discussion  in  Sections  3.1.1  and  3.1.2. 


3.2.1.  Solution  for  the  viscoelasticitc  system 

The  analysis  can  be  considerably  simplified  when  the  following 
conditions  are  fulfilled  [42]: 


dky 


(4) 


3.1.4.  Stress-strain  relation 

In  general,  the  elastic  modulus  E  was  used  to  construct  the  defor¬ 
mation  model.  The  relation  between  the  elastic  modulus  E  and  the 
shear  modulus  G  was  calculated  using  Poisson’s  ratio  v  as  in  the 
following  equation: 

E  =  2(1  +  v)G  (5) 

In  the  experiment  using  the  rheometer,  only  the  shear  modu¬ 
lus  was  loaded  on  the  test  material.  Flowever,  in  the  situation  of 
deformation  simulation,  the  stress  state  is  more  complex  because 
it  has  more  moduli.  We  assumed  that  the  nonlinearity  of  the  elastic 
modulus  would  be  decided  by  the  relative  strain  calculated  in  (6). 


£r 


{(£i  -s2f  +  (e2  -  S3)2  +(£3 


Si)2) 


(6) 


where  E  is  the  elastic  modulus,  v  is  Poisson’s  ratio,  £r  is  relative 
strain,  s ?,  e2,  and  £3  are  the  principal  strains. 

Thus,  from  these  considerations,  the  material  properties  using 
the  liver  model  can  be  described  in  (7)  and  (8). 


E{£^=a 


(7) 


E(£  r) 


E0  (er  <  sQ) 

F0(l  +  Ctfe(£ r  —  £q)  )  (£r  >  £0) 


(8) 


where  E(£r)  is  nonlinear  elastic  modulus,  £r  is  relative  strain,  s  is 
strain,  o  is  stress,  E0  is  the  elastic  modulus  between  linear  rang,  ots  is 
the  coefficient  deciding  the  change  of  stiffness,  and  £0  is  the  strain  at 
which  the  characteristics  of  the  liver  change  to  show  nonlinearity. 


-  The  derivative  operator  of  (9)  is  a  common  factor  in  all  element 
stiffness. 

-  Only  the  external  loads  influence  the  stresses. 

Then,  Eq.  (12)  is  derived  from  (9). 

K(U)U  =  F'(F'  =  dL-®F)  (12) 

where  is  the  fcth-order  integration. 

Eq.  (12)  is  identical  to  the  elastic  problem  when  the  virtual 
external  force  vector  F  is  used.  The  fractional  calculation  (12)  for 
each  component  of  the  external  force  vector  F  was  implemented 
to  obtain  the  virtual  external  force  vector  F.  We  used  the  sampling 
time  scaling  property  introduced  in  [43]  to  make  a  discrete  frac¬ 
tional  calculation.  We  used  the  sampling  time  scaling  property  to 
consider  the  discrete  fractional  order  integrals  as  the  “deformation” 
of  their  integer  order  counterparts. 

3.2.2.  Nonlinear  system 

Incremental  approaches  are  important  to  obtain  a  significant 
answer,  because  the  answer  is  not  unique  for  many  nonlinear  sit¬ 
uations.  The  incremental  form  of  a  discrete  nonlinear  model  is 
generally  written  as  in  (13). 

Kt(Un)AUn  =  AFn  (13) 

where  n  is  step  of  calculation,  Kt  is  the  tangential  stiffness 
matrix,  Un  is  the  overall  displacement  vector,  AUn  is  an  increment 
of  the  overall  displacement  vector,  and  AFn  is  an  increment  of  the 
overall  external  force.  The  tangential  matrix  Kt  is  described  by  (14) 
from  Eq.  (11). 

Kt(U)=  Y  kt ^  (14) 

all  elements 


Y.  Kobayashi  et  al.  /  Computerized  Medical  Imaging  and  Graphics  34  (2010)  9-18 


15 


(a)  Experimental  apparatus  (b)  Ultrasound  image 


Fig.  9.  Experimental  setup  of  in  vivo  experiment.  The  inferior  vena  cava  of  the  liver  was  chosen  as  the  virtual  target  of  the  experiment.  The  needle  insertion  manipulator  with 
ultrasound  probe  was  set  using  the  endoscope  holding  device  at  the  location  to  visualize  the  target  inferior  vena  cava  in  the  ultrasound  image.  Then,  the  surgeon  ordered 
the  needle  insertion  path  from  the  computer.  The  needle  was  inserted  into  the  porcine  liver  at  a  constant  speed  of  5  mm/s  from  the  path,  (a)  Experimental  apparatus  and  (b) 
ultrasound  image. 


kt{Sr)  = 


fco 


(fir  <  So) 


[1  +ae(£r-£o)  +2a£(£r-£o)Sr]ko  (£r  >  £0) 


(15) 


We  used  both  the  Euler  method  and  the  modified 
Newton-Raphson  method  to  solve  the  nonlinear  system. 


3.2.3.  Calculation  process 

Based  on  these  discussions,  the  solution  for  the  system  of  (9) 
can  be  described  as  follows.  First,  the  virtual  external  force  F  was 
calculated;  then  the  incremental  of  F  (AF)  was  computed.  The 
solution  to  the  nonlinear  system  described  in  Section  3.2.2  was  then 
carried  out  using  AF'. 


Kt(Un)AUn  =  AFn  (16) 

where  n  is  step  of  calculation,  Kt  is  the  tangential  stiffness  matrix, 
Un  is  the  overall  displacement  vector,  AUn  is  an  increment  of  the 
overall  displacement  vector,  and  AFn  is  an  increment  of  the  virtual 
external  force. 

4.  Validation  experiment 

4.1.  Overview 

This  section  shows  the  in  vivo  experimental  validation  of  our 
needle  insertion  manipulator  and  physical  liver  model.  The  needle 
insertion  experiment  was  carried  out  on  a  live  porcine  liver.  The 
deformation  of  this  liver  was  obtained  from  ultrasound  images, 
while  the  deformation  of  the  liver  model  was  simulated  by  the 
computer  program.  By  comparing  the  relations  between  needle  dis¬ 
placement  and  force  on  the  real  liver  and  those  in  the  liver  model, 
we  also  were  able  to  evaluate  the  model. 

The  following  experiment  in  Section  4.2.1  is  the  same  as  the 
experiment  described  in  our  previous  paper  [40].  This  paper 
presents  the  validation  using  the  model  with  porcine  liver  shape, 
while  the  liver  model  in  [40]  was  validated  with  the  assumed  that 
the  liver  was  rectangle  shape. 

4.2.  Methods 

4.2.1.  In  vivo  experiment 

Because  a  healthy  porcine  was  used  in  the  experiment,  the  infe¬ 
rior  vena  cava  of  the  liver  was  chosen  as  the  virtual  target  of  the 
experiment.  An  inferior  vena  cava  of  a  liver  is  clearly  visible  by 
ultrasound  image  and  it  is  easy  to  confirm  the  phenomenon  dur¬ 
ing  needle  insertion.  Fig.  9  shows  the  experimental  setup  of  the 
in  vivo  experiment.  First,  the  surgeon  searched  the  target  inferior 


vena  cava  by  manipulating  the  ultrasound  probe  attached  to  the 
needle  insertion  manipulator.  Next,  the  surgeon  fixed  the  probe 
with  the  needle  insertion  manipulator  using  the  endoscope  hold¬ 
ing  device  at  the  location  to  visualize  the  target  inferior  vena  cava 
in  the  ultrasound  image.  Then,  the  surgeon  then  used  the  graphi¬ 
cal  user  interface  (GUI)  to  order  the  needle  insertion  path  from  the 
computer.  Then,  the  needle  was  inserted  into  the  porcine  liver  at 
a  constant  speed  of  5  mm/s  from  the  path,  the  ultrasound  image 
was  captured,  and  the  force  of  the  tissues  against  the  needle  was 
measured  from  force-sensor  data. 

4.2.2.  Analysis  using  liver  model 

To  create  the  shape  of  the  liver  model,  we  used  MRI  image  of 
porcine  liver  (Fig.  10).  We  have  not  yet  integrated  the  registration 
technique  with  preoperative  shape  image  of  liver  such  as  described 
in  Section  2.1,  so  we  used  the  MRI  image  obtained  from  other 
porcine  in  preliminary  experiment.  It  means  that  the  size  of  liver 
model  was  fit  to  the  real  liver  while  the  shape  was  decided  refer¬ 
ring  to  MRI  image  of  other  porcine  liver  shown  in  Fig.  1 0.  Therefore, 
this  validation  experiment  was  not  rigorous,  because  the  shape  of  a 
real  porcine  liver  used  in  this  experiment  was  not  exactly  the  same 
as  that  of  porcine  liver  of  MRI  image.  A  more-detailed  analysis  and 
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Fig.  10.  Physical  model  of  porcine  liver,  (a)  MRI  image  and  (b)  physical  model.  We 
used  MRI  image  of  porcine  liver  to  create  the  shape  of  the  liver  model.  Applying 
Delaunay’s  method  to  the  extracted  outline  data,  a  FEM  mesh  was  created  for  use  in 
the  computer  simulation.  The  green-colored  nodes  represent  the  fixed  ends,  taking 
into  consideration  rib  and  spine  restrictions  at  the  dorsal  aspect  of  the  abdomen, 
and  the  red  node  identifies  the  target  position.  The  liver  was  assessed  as  though  it 
were  uniform. 
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(a)  Ultrasound  Image 


(b)  Simulation 
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Fig.  11.  Deformation  of  liver  tissue  by  needle  insertion,  (a)  As  shown  in  ultrasound  images  and  (b)  as  shown  in  simulation.  The  inferior  vena  cava  in  the  liver  collapsed  from 
the  force  put  on  the  needle  by  the  liver  tissues  during  insertion,  (b)  Shows  a  simulation  of  the  deformation  of  the  liver,  which  is  confirmed  by  the  ultrasound  images  captured 
during  the  experiment.  In  (b),  the  color  of  each  finite  element  indicates  the  stress  on  it. 


evaluation  of  the  liver  model  with  the  real  shape  of  a  porcine  liver 
used  in  the  experiment  should  be  carried  out  in  future. 

The  images  were  outlined,  and  the  outlines  were  manually 
extracted.  Applying  Delaunay’s  method  to  the  extracted  outline 
data,  a  FEM  mesh  was  created  for  use  in  the  computer  simulation 
(Fig.  10).  In  Fig.  10,  the  green-colored  nodes  represent  the  fixed 
ends,  taking  into  consideration  rib  and  spine  restrictions  at  the  dor¬ 
sal  aspect  of  the  abdomen,  and  the  red  node  identifies  the  target 
position. 

It  also  should  be  noted  that  the  liver  was  assessed  as  though  it 
were  uniform,  whereas  an  actual  liver  is  made  up  of  non-uniform 
tissue  that  even  can  include  cirrhosis  and  cancer  cells.  The  stiffness 
parameters,  such  as  E0,  s0  and  ae ,  of  the  normal  tissue  were  set 
manually  to  fit  the  measured  data  of  needle  insertion  force.  There 
also  are  large  differences  between  the  stiffness  parameters  of  the 
in  vitro  liver  and  the  in  vivo  liver.  It  is  said  that  the  in  vivo  liver  is 
harder  than  the  in  vitro  liver.  It  is  assumed  in  our  modeling  that 
the  viscoelastic  and  nonlinear  properties  of  in  vitro  liver  and  in 
vivo  liver  can  be  represented  using  same  Eq.  (4)  and  only  stiffness 
parameter  such  as  E0, 80  and  as ,  are  different  between  in  vitro  liver 
and  in  vivo  liver. 

The  needle  was  assumed  to  be  inserted  at  a  constant  speed  of 
5  mm/s.  The  time-series  data  of  needle  displacement  and  target  dis¬ 
placement  were  collected  during  the  experiment.  Since  only  the 
initial  loading  phase  was  analyzed  in  this  simulation,  a  simula¬ 
tion  including  puncture  phenomena  will  be  carried  out  in  the  near 
future. 

4.3.  Results  and  discussion 

4.3.1.  Needle  insertion  manipulator 

The  ultrasound  images  in  Fig.  11(a)  show  that  the  inferior  vena 
cava  in  the  liver  collapsed  from  the  force  put  on  the  needle  by  the 
liver  tissues  during  insertion.  Finally,  the  needle  tip  reached  the 
target  in  the  inferior  vena  cava  (not  shown  in  figure). 

Fig.  12  shows  the  relation  between  needle  displacement  and 
force  during  the  experiment.  As  can  be  seen  in  the  figure,  there  is  a 
nonlinear  increase  in  force  (of  the  liver  tissues  against  the  needle) 
as  a  function  of  increased  needle  displacement  (distance  traveled 
by  the  needle).  As  also  shown  in  previous  work  [34,37],  Fig.  12 
shows  that  the  main  puncture  event  is  designated  by  a  peak  in 


force  after  a  steady  rise,  followed  by  a  sharp  decrease.  This  data 
explains  that  when  the  needle  punctures  the  liver,  the  tissues  do 
not  respond  with  a  constant  force.  Instead,  the  needle  first  pushes 
the  tissue  and  then  instantaneously  punctures  it.  The  first  sharp 
decrease  in  the  graph  of  Fig.  12  means  that  the  needle  punctured 
the  liver  membrane,  while  the  second  sharp  decrease  means  that 
the  needle  punctured  the  inferior  vena  cava.  In  addition,  after  the 
experiment,  we  dissected  the  liver  and  confirmed  that  the  needle 
had  punctured  the  inferior  vena  cava.  Both  the  ultrasound  images 
and  force  information  verified  the  accuracy  and  effectiveness  of  our 
needle  insertion  manipulator. 

The  material  properties  of  a  hard  cancer  are  different  from  a 
soft  inferior  vena  cava,  which  is  virtual  target  of  this  experiment. 
Then,  we  should  conduct  the  experiment,  in  which  the  target  is 
a  cancer  and/or  other  hard  tissue,  to  validate  our  manipulator,  in 
future  work. 

4.3.2.  Physical  liver  model 

Fig.  11(b)  shows  a  simulation  of  the  deformation  of  the  liver, 
which  is  confirmed  by  the  ultrasound  images  captured  during  the  in 
vivo  experiment  that  are  shown  in  Fig.  11(a).  In  Fig.  11(b),  the  color 


Fig.  12.  Force  as  a  function  of  needle  displacement.  The  first  sharp  decrease  in  the 
experimental  result  means  that  the  needle  punctured  the  liver  membrane,  while 
the  second  sharp  decrease  means  that  the  needle  punctured  the  inferior  vena  cava. 
The  simulation  result  present  that  the  liver  model  reproduces  the  relation  between 
needle  displacement  and  force  observed  in  the  live  liver  with  high  accuracy  during 
only  the  initial  pushing  phase. 
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of  each  finite  element  indicates  the  stress  on  it.  Fig.  12  shows  the 
relation  between  needle  displacement  and  force  for  both  the  real 
liver  of  the  in  vivo  experiment  and  the  liver  of  the  simulation  model. 
As  is  evident  in  the  figure,  the  liver  model  we  developed  reproduces 
the  relation  between  needle  displacement  and  force  observed  in  the 
live  liver  with  high  accuracy  during  only  the  initial  pushing  phase. 
In  summary,  it  is  strongly  suggested  from  the  discussion  above  that 
the  liver  model  we  developed  accurately  reproduces  the  in  vivo 
situation. 

However,  our  validation  using  this  liver  shape  was  not  rigorous, 
because  the  shape  of  a  real  porcine  liver  is  not  the  same  as  that  of 
liver  model.  Therefore,  the  deformation  analysis  and  evaluation  of 
a  liver  model  with  the  shape  of  a  real  porcine  liver  should  be  carried 
out  in  the  future. 

Moreover,  only  the  initial  loading  phase  was  analyzed  in  this 
numerical  simulation.  It  is  necessary  to  conduct  simulation  that 
includes  an  analysis  of  the  situation  in  which  the  needle  advances 
into  the  liver,  so  that  we  provide  the  further  detailed  information 
to  a  doctor.  A  simulation  including  puncture  phenomenon  will  be 
carried  out  in  the  near  future. 

5.  Conclusion 

The  purpose  of  our  research  was  to  develop  an  integrated 
robotic  system  with  image  guidance  and  deformation  simulation 
for  the  purpose  of  accurate  needle  insertion.  We  designed  both 
an  ultrasound-guided  needle  insertion  manipulator  and  a  physi¬ 
cal  model  to  simulate  liver  deformation.  We  carried  out  an  in  vivo 
experiment  using  a  porcine  liver  to  validate  the  effectiveness  of  our 
needle  insertion  manipulator  and  liver  model.  The  results  of  the 
in  vivo  experiment  showed  that  the  needle  insertion  manipulator 
accurately  positions  the  needle  tip  into  the  target.  The  experimental 
results  also  showed  that  the  liver  model  accurately  reproduced  the 
nonlinear  increase  of  force  upon  the  needle  during  insertion.  Based 
on  these  results,  it  is  suggested  that  the  needle  insertion  manipula¬ 
tor  and  physical  liver  model  developed  and  validated  in  this  work 
are  effective  for  accurate  needle  insertion. 

In  future,  further  precise  organ  modeling  and  validation  will 
be  carried  out.  For  example,  a  modeling  based  on  acquisition  of 
organ  geometries  using  CT  or  MRI  images  will  be  researched  for 
actual  applications.  In  addition,  the  needle  insertion  manipulator 
and  physical  liver  model  will  be  further  integrated  as  a  system  to 
achieve  safe  and  precise  operation. 
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